Skip to content

fold U+0345 after canonical ordering in utf8proc_decompose_custom - #379

Open
shamsManna wants to merge 1 commit into
JuliaStrings:masterfrom
shamsManna:ypogegrammeni-casefold
Open

shamsManna wants to merge 1 commit into
JuliaStrings:masterfrom
shamsManna:ypogegrammeni-casefold

Conversation

@shamsManna

Copy link
Copy Markdown
Contributor

utf8proc_decompose_custom case-folds each codepoint before the canonical ordering, but U+0345 (ccc 240) folds to U+03B9, a starter, so marks after it end up on the iota instead of the base letter and canonically equivalent strings fold differently: utf8proc_NFKC_Casefold maps U+1FB3 U+0301 to U+03B1 U+03AF but its NFC form U+1FB4 to U+03AC U+03B9 (fixes #3). this leaves U+0345 unfolded, also inside the 63 precomposed letters that decompose to it, and folds it once its run of marks ends at a starter of the canonical decomposition (dropped ignorables count), so the output matches the UCD's NFC(NFKC_CF(NFD(x))) on every codepoint and on 750k random strings and is byte-identical to before for input without U+0345. CHARBOUND is left as it was since its sentinels block the reordering, and there's a regression test in test/misc.c.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

COMBINING GREEK YPOGEGRAMMENI case-folding

1 participant